Passage feedback with IRIS

In our TREC-8 interactive experiment, we compared the effectiveness of a user-defined passage feedback system with a conventional document feedback system. The passage feedback system (pf) employed user-defined passages to formulate a feedback query, while the document feedback (df) system utilized document-level relevance judgments. The df system employed the adaptive linear model for retrieval, and pf used weighted query expansion based on positive and negative feedback. Twenty-four searchers performed the same 6 search tasks (3 per system) in varying search and system order in accordance with TREC-8 guidelines. We hypothesized that the pf system, which featured interactive query expansion by user-selected passages, would have an advantage over the df system, which relied on automatic query expansion based on relevance judgments of whole documents. Our initial analysis of experiment results appeared to reject this hypothesis, as the document feedback system showed slightly higher overall performance statistics than the passage feedback system. A closer look, however, gave a different impression. When we examined the results by the system-order groups (i.e. df1, pf1, df2, pf2, where 1 signifies “first system used”, and 2 signifies “second system used”), we saw no significant performance difference in systems except for the first used passage feedback system (pf1) that showed noticeably poor results. Analysis of these results as well as the examination of transaction logs and questionnaire data suggest to us that the passage feedback system might have been more difficult to learn than the document feedback system. It is possible that the difficulty of learning to use the passage feedback system resulted in the poor performance of the first used passage feedback system. Though the overall results are inconclusive, we are encouraged by the competitive performance of the second passage feedback system use. If the performance improvement of the passage feedback system over time is indeed affected by